\documentclass[10pt]{article}
\usepackage{fullpage}
\parindent0in
\parskip0.1in

\def \memcpy  {{\tt upc\_memcpy}}
\def \memget  {{\tt upc\_memget}}
\def \memput  {{\tt upc\_memput}}
\def \memset  {{\tt upc\_memset}}

\def \memall  {{\tt upc\_mem\{put, get, cpy\}}}
\def \memstar {{\tt upc\_mem*}}

\def \gsync  {{\tt upc\_gsync[\_attempt]}}
\def \gsynci {{\tt upc\_gsynci[\_attempt]}}
\def \lsync  {{\tt upc\_lsync[\_attempt]}}
\def \lsynci {{\tt upc\_lsynci[\_attempt]}}

\def \complete {{\tt UPC\_COMPLETE\_HANDLE}}

\title{UPC Asynchronous Memory Copy Library Extensions Proposal\\Version 1.1}
\author{Troy Johnson\\Cray Inc. \and Steve Vormwald\\Cray Inc.}

\begin{document}

\maketitle

\begin{abstract}
This document examines limitations of the existing UPC~1.2 memory copy library
functions, reviews how various unofficial library extensions address the
limitations, and proposes official library extensions.  The description of each
proposed function is in the same format as existing UPC~1.2 library function
specifications for ease of consideration and adoption.  This proposal is an
alternative to the ``Proposal for Extending the UPC Memory Copy Library
Functions,'' Lawrence Berkeley National Lab Tech Report LBNL-56495, and offers
different semantics that are more compatible with the UPC~1.2 memory model.
\end{abstract}

\section{Introduction}

Cray identifies two major limitations to the UPC~1.2 copy functions.  This
section describes these limitations and then discusses the Berkeley UPC and Cray
UPC extensions, including how they address each limitation.  Neither UPC
implementation completely removes both limitations while remaining fully
compatible with the UPC~1.2 memory model.  Therefore, this section motivates a
new proposal that addresses both limitations in a compatible manner.

\subsection{UPC~1.2 References}

Statements in this section are based on four specific paragraphs of UPC~1.2.
For convenience, these paragraphs are reproduced below.

\paragraph{5.1.2.3.5} When a thread's program order dictates a set of relaxed
operations followed by a strict operation, all threads will observe the effects
of the prior relaxed operations made by the issuing thread (in some order)
before observing the strict operation. Similarly, when a thread's program order
dictates a strict access followed by a set of relaxed accesses, the strict
access will be observed by all threads before any of the subsequent relaxed
accesses by the issuing thread. Consequently, strict operations can be used to
synchronize the execution of different threads, and to prevent the apparent
reordering of surrounding relaxed operations across a strict operation.

\paragraph{6.6.1.5} The upc fence statement is equivalent to a null strict
access.  This insures~[{\em sic}] that all shared accesses issued before the
fence are complete before any after it are issued.

\paragraph{B.3.2.1.1} For non-collective functions in the UPC standard library
(e.g. \memall{}), any implied data accesses to shared objects behave as a set of
relaxed shared reads and relaxed shared writes of unspecified size and ordering,
issued by the calling thread. No strict operations or fences are implied by a
non-collective library function call, unless explicitly noted otherwise.

\paragraph{B.3.2.1.3} $\ldots$ Explicit use of upc fence immediately preceding
and following non-collective library calls operating on shared objects is the
recommended method for ensuring ordering with respect to surrounding relaxed
operations issued by the calling thread, in cases where such ordering guarantees
are required for program correctness.

\subsection{UPC~1.2 Limitation~\#1 -- Cannot Detect Individual Global Visibility}

Various algorithms require the ability to deposit data in memory with affinity
to a different thread and then signal that thread that all of the data has
arrived.  Based on the UPC~1.2 paragraphs above, the following code illustrates
how this idiom can be coded:

\begin{verbatim}
    upc_memput( dst, src, n );
    *ptr_to_strict = 1; // upc_threadof( ptr_to_strict ) == upc_threadof( dst )
\end{verbatim}

The problem with this code is that it forces all accesses prior to the strict
Put to become globally visible, not just the collection of relaxed writes within
the \memput{} call.  For example, the programmer may wish to overlap multiple
\memput{} calls and notify each target thread, but the act of notifying {\em
any} target thread forces global visibility of {\em every} \memput{} call.

\subsection{UPC~1.2 Limitation~\#2 -- Cannot Overlap Work with Local Visibility}

The \memstar{} functions may return prior to their results being made globally
visible, but they must maintain program order with respect to other accesses
made by the calling thread by ensuring local visibility.  Using \memput{} as an
example, any subsequent modification of the source data by the calling thread
must not affect the result.  Also, if the calling thread Gets back the data that
it just Put, it must see the correct data.  The \memstar{} functions cannot
return until such local visibility is achieved.  Therefore, the programmer has
no option for initiating a copy, doing unrelated work, ensuring local
visibility, and then doing work that may access the same data (e.g., reusing a
source buffer) before waiting on global visibility.

An implementation may or may not distinguish local visibility from global
visibility; the latter implies the former, so it is always safe for an
implementation to wait for global visibility in order to provide local
visibility.  For an implementation that does not make the distinction, this
limitation is not as important as Limitation~\#1.  For implementations that
distinguish the two kinds of visibility, this limitation's importance to the
programmer depends on whether significant time elapses between initiating the
copy and achieving local visibility, and between achieving local and global
visibility.

\section{Current Language Extensions} 

\subsection{Berkeley UPC}

BUPC provides \_async and \_asynci variations of the existing \memstar{} copy
functions.  These copies proceed independently of any existing synchronization
mechanism (e.g., upc\_fence, strict accesses, barriers)~\footnote{This design is
incompatible with the UPC~1.2 memory model.} and come with their own
synchronization functions that ensure global visibility.  These extensions
address Limitation~\#1.

An \_async call plus a BUPC waitsync call provides a stronger guarantee, global
visibility, than an existing \memstar{} call, which provides local visibility.
Therefore, these extensions do not address Limitation~\#2.

\subsection{Cray UPC}

Cray UPC provides \_nb and \_nbi variations of the existing \memstar{} copy
functions.  These copies become globally visible following either a fence
(strict access) or a call to a synchronization function.  Limitation~\#1 is
removed, but only if a relaxed access is used for each completion
notification~\footnote{The use of a relaxed access as a form of thread
synchronization conflicts with the UPC~1.2 memory model, but can be resolved by
adding some fence-like properties to the synchronization function.}

Cray UPC is an example of an implementation that does not distinguish between
local and global visibility.  Therefore, there are no Cray extensions for
detecting local visibility and Limitation~\#2 is not addressed.

\section{Cray Proposal for Asynchronous Memory Copy Functions}

\subsection{Summary}

Cray proposes to address both Limitation~\#1 and Limitation~\#2 using the
following approach.  Each \memstar{} function will have an \_async and an
\_asynci variation, similar to the BUPC extensions.   The \_async functions give
the programmer explicit control and responsibility to manage completion of each
copy.  The handle returned by an \_async function can be passed to a \lsync{} or
a \gsync{} function to ensure local or global visibility of the associated copy.
Checking the handle for local visibility is optional, but successfully checking
the handle for global visibility is required~\footnote{Recall that global
visibility implies local visibility.} so that an implementation can free any
resources that may be associated with the handle.  Furthermore, the successful
\gsync{} must occur prior to the next fence (strict access), otherwise after the
fence the copy's global visibility and the handle's validity are undefined.
These restrictions maintain correct fence semantics without requiring an
implementation to internally track every handle that it issues, which it would
need to do if the \gsync{} was optional and the fence had to synchronize all
unsynchronized handles.

The \_asynci functions allow the programmer to optionally manage completion of a
single, implicit group of copies that becomes globally visible at the next
fence.  Checking either local or global visibility is optional for the group of
\_asynci copies.  Unlike an \_async copy, a fence will synchronize any
unsynchronized \_asynci copies.  This requirement is not expected to burden
implementations because all \_asynci copies can be tracked together.

A successful \gsync{} or \gsynci{} call has ``half-fence'' semantics such that a
relaxed access issued before it may be observed after it, but no relaxed access
issued after it may be observed before it.  The former allows multiple \_async
operations to overlap even if one of them is synchronized and the latter allows
a relaxed access to be used for completion notification.  A successful \lsync{}
or \lsynci{} call has no effect on the observation of accesses, but accesses to
the source and destination buffers are prohibited prior to the call.

The remainder of this section specifies each proposed function in detail.  The
intent is for the same language to be copied into the UPC specification if the
proposal is accepted.

\subsection{Memory Copy Handle Constraints}
\setcounter{secnumdepth}{4}

\paragraph{} An implementation shall define the following type and value:

\begin{verbatim}
type  upc_handle_t
value UPC_COMPLETE_HANDLE
\end{verbatim}

\paragraph{} \complete{} shall have type upc\_handle\_t.  All of its bits shall
be 0.

\paragraph{} A handle is valid only on the thread which originally obtained it
from an \_async call.  It is an error to use the handle for any purpose on a
different thread.

\paragraph{} A handle shall be passed to a successful \gsync{} call.  It is an
error to discard a handle and never globally synchronize it unless the handle
has value \complete{}. 

\paragraph{} A handle is {\em not} required to be passed to a successful
\lsync{} call.

\paragraph{} Once a handle is globally synchronized, it becomes invalid and
shall not be used for any purpose.

\newpage
\subsection{upc\_memcpy\_async}
\def\function{{\tt upc\_memcpy\_async}}

\subsubsection{Synopsis}

\begin{verbatim}
upc_handle_t upc_memcpy_async( shared void * restrict dst,
                               shared const void * restrict src, size_t n );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The parameters shall be treated identically to the parameters of
\memcpy{}.

\paragraph{} The \function{} function shall have the same effects as \memcpy{}.
If the returned handle == \complete{}, then the effects shall be globally
visible; otherwise, the effects shall become locally visible after passing the
returned handle to a successful \lsync{} call and shall become globally visible
after passing the returned handle to a successful \gsync{} call.

\paragraph{} An implementation shall choose whether to return \complete{} or to
perform an asynchronous copy.  For example, an implementation might choose to
perform a synchronous copy if upc\_threadof(~dst~) == upc\_threadof(~src~) ==
MYTHREAD.  Implementations are encouraged to perform asynchronous copies and
return quickly whenever a synchronous copy would take significant time.

\paragraph{} It is an error to access the memory pointed to by dst or src until
the copy is locally visible.

\paragraph{} The following two code sequences show the relationship between
\memcpy{} and \function{}:

\begin{verbatim}
upc_memcpy( dst, src, n ); // effects are locally visible after return
...                        // code that may access targets of dst and src
upc_fence;                 
...                        // effects are globally visible, but also all
                           //   prior accesses are globally visible


upc_handle_t handle = upc_memcpy_async( dst, src, n );
...                   // code that may not access targets of dst and src
upc_lsync( &handle ); // effects are locally visible after return (optional call)
...                   // code that may access targets of dst and src (requires lsync)
upc_gsync( &handle ); // effects are globally visible after return (required call
                      //     prior to fence unless handle == UPC_COMPLETE_HANDLE)
upc_fence;
...                   // all prior accesses are globally visible
\end{verbatim}

\newpage
\subsection{upc\_memget\_async}
\def\function{{\tt upc\_memget\_async}}

\subsubsection{Synopsis}

\begin{verbatim}
upc_handle_t upc_memget_async( void * restrict dst,
                               shared const void * restrict src, size_t n );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The parameters shall be treated identically to the parameters of
\memget{}.

\paragraph{} The \function{} function shall have the same effects as \memget{}.
If the returned handle == \complete{}, then the effects shall be globally
visible; otherwise, the effects shall become locally visible after passing the
returned handle to a successful \lsync{} call and shall become globally visible
after passing the returned handle to a successful \gsync{} call.  

\paragraph{} An implementation shall choose whether to return \complete{} or to
perform an asynchronous copy.  For example, an implementation might choose to
perform a synchronous copy if upc\_threadof(~src~) == MYTHREAD.  Implementations
are encouraged to perform asynchronous copies and return quickly whenever a
synchronous copy would take significant time.

\paragraph{} It is an error to access the memory pointed to by dst or src until
the copy is locally visible.

\paragraph{} The following two code sequences show the relationship between
\memget{} and \function{}:

\begin{verbatim}
upc_memget( dst, src, n ); // effects are locally visible after return
...                        // code that may access targets of dst and src
upc_fence;                 
...                        // effects are globally visible, but also all
                           //   prior accesses are globally visible


upc_handle_t handle = upc_memget_async( dst, src, n );
...                   // code that may not access targets of dst and src
upc_lsync( &handle ); // effects are locally visible after return (optional call)
...                   // code that may access targets of dst and src (requires lsync)
upc_gsync( &handle ); // effects are globally visible after return (required call
                      //     prior to fence unless handle == UPC_COMPLETE_HANDLE)
upc_fence;
...                   // all prior accesses are globally visible
\end{verbatim}

\newpage
\subsection{upc\_memput\_async}
\def\function{{\tt upc\_memput\_async}}

\subsubsection{Synopsis}

\begin{verbatim}
upc_handle_t upc_memput_async( shared void * restrict dst,
                               const void * restrict src, size_t n );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The parameters shall be treated identically to the parameters of
\memput{}.

\paragraph{} The \function{} function shall have the same effects as \memput{}.
If the returned handle == \complete{}, then the effects shall be globally
visible; otherwise, the effects shall become locally visible after passing the
returned handle to a successful \lsync{} call and shall become globally visible
after passing the returned handle to a successful \gsync{} call.

\paragraph{} An implementation shall choose whether to return \complete{} or to
perform an asynchronous copy.  For example, an implementation might choose to
perform a synchronous copy if upc\_threadof(~dst~) == MYTHREAD.  Implementations
are encouraged to perform asynchronous copies and return quickly whenever a
synchronous copy would take significant time.

\paragraph{} It is an error to access the memory pointed to by dst or src until
the copy is locally visible.

\paragraph{} The following two code sequences show the relationship between
\memput{} and \function{}:

\begin{verbatim}
upc_memput( dst, src, n ); // effects are locally visible after return
...                        // code that may access targets of dst and src
upc_fence;                 
...                        // effects are globally visible, but also all
                           //   prior accesses are globally visible


upc_handle_t handle = upc_memput_async( dst, src, n );
...                   // code that may not access targets of dst and src
upc_lsync( &handle ); // effects are locally visible after return (optional call)
...                   // code that may access targets of dst and src (requires lsync)
upc_gsync( &handle ); // effects are globally visible after return (required call
                      //     prior to fence unless handle == UPC_COMPLETE_HANDLE)
upc_fence;
...                   // all prior accesses are globally visible
\end{verbatim}

\newpage
\subsection{upc\_memset\_async}
\def\function{{\tt upc\_memset\_async}}

\subsubsection{Synopsis}

\begin{verbatim}
upc_handle_t upc_memset_async( shared void *dst, int c, size_t n );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The parameters shall be treated identically to the parameters of
\memset{}.

\paragraph{} The \function{} function shall have the same effects as \memset{}.
If the returned handle == \complete{}, then the effects shall be globally
visible; otherwise, the effects shall become locally visible after passing the
returned handle to a successful \lsync{} call and shall become globally visible
after passing the returned handle to a successful \gsync{} call.

\paragraph{} An implementation shall choose whether to return \complete{} or to
perform an asynchronous copy.  For example, an implementation might choose to
perform a synchronous copy if upc\_threadof(~dst~) == MYTHREAD.  Implementations
are encouraged to perform asynchronous copies and return quickly whenever a
synchronous copy would take significant time.

\paragraph{} It is an error to access the memory pointed to by dst until
the copy is locally visible.

\paragraph{} The following two code sequences show the relationship between
\memset{} and \function{}:

\begin{verbatim}
upc_memset( dst, c, n );   // effects are locally visible after return
...                        // code that may access target of dst
upc_fence;                 
...                        // effects are globally visible, but also all
                           //   prior accesses are globally visible


upc_handle_t handle = upc_memset_async( dst, c, n );
...                   // code that may not access targets of dst
upc_lsync( &handle ); // effects are locally visible after return (optional call)
...                   // code that may access target of dst (requires lsync)
upc_gsync( &handle ); // effects are globally visible after return (required call
                      //     prior to fence unless handle == UPC_COMPLETE_HANDLE)
upc_fence;
...                   // all prior accesses are globally visible
\end{verbatim}

\newpage
\subsection{upc\_memcpy\_asynci}
\def\function{{\tt upc\_memcpy\_asynci}}

\subsubsection{Synopsis}

\begin{verbatim}
void upc_memcpy_asynci( shared void * restrict dst,
                        shared const void * restrict src, size_t n );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The parameters shall be treated identically to the parameters of
\memcpy{}.

\paragraph{} The \function{} function shall have the same effects as \memcpy{}.
The effects of all prior \function{} calls shall become locally visible after a
successful \lsynci{} call and shall become globally visible after either a
successful \gsynci{} call or a fence (strict access).

\paragraph{} It is an error to access the memory pointed to by dst or src until
the copy is locally visible.

\paragraph{} The following two code sequences show the relationship between
\memcpy{} and \function{}:

\begin{verbatim}
upc_memcpy( dst, src, n ); // effects are locally visible after return
...                        // code that may access targets of dst and src
upc_fence;                 
...                        // effects are globally visible, but also all
                           //   prior accesses are globally visible


upc_memcpy_asynci( dst1, src1, n );
upc_memcpy_asynci( dst2, src2, n );
...           // code that may not access targets of dst1, dst2, src1, and src2
upc_lsynci(); // effects are locally visible after return (optional call)
...           // code that may access targets of dst1, dst2, src1, and src2 (requires lsynci)
upc_gsynci(); // effects are globally visible after return (optional call)
...
upc_fence;
...           // all prior accesses are globally visible
\end{verbatim}

\newpage
\subsection{upc\_memget\_asynci}
\def\function{{\tt upc\_memget\_asynci}}

\subsubsection{Synopsis}

\begin{verbatim}
void upc_memget_asynci( void *dst, shared const void *src, size_t n );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The parameters shall be treated identically to the parameters of
\memget{}.

\paragraph{} The \function{} function shall have the same effects as \memget{}.
The effects of all prior \function{} calls shall become locally visible after a
successful \lsynci{} call and shall become globally visible after either a
successful \gsynci{} call or a fence (strict access).

\paragraph{} It is an error to access the memory pointed to by dst or src until
the copy is locally visible.

\paragraph{} The following two code sequences show the relationship between
\memget{} and \function{}:

\begin{verbatim}
upc_memget( dst, src, n ); // effects are locally visible after return
...                        // code that may access targets of dst and src
upc_fence;                 
...                        // effects are globally visible, but also all
                           //   prior accesses are globally visible


upc_memget_asynci( dst1, src1, n );
upc_memget_asynci( dst2, src2, n );
...           // code that may not access targets of dst1, dst2, src1, and src2
upc_lsynci(); // effects are locally visible after return (optional call)
...           // code that may access targets of dst1, dst2, src1, and src2 (requires lsynci)
upc_gsynci(); // effects are globally visible after return (optional call)
...
upc_fence;
...           // all prior accesses are globally visible
\end{verbatim}

\newpage
\subsection{upc\_memput\_asynci}
\def\function{{\tt upc\_memput\_asynci}}

\subsubsection{Synopsis}

\begin{verbatim}
void upc_memput_asynci( shared void *dst, const void *src, size_t n );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The parameters shall be treated identically to the parameters of
\memput{}.

\paragraph{} The \function{} function shall have the same effects as \memput{}.
The effects of all prior \function{} calls shall become locally visible after a
successful \lsynci{} call and shall become globally visible after either a
successful \gsynci{} call or a fence (strict access).

\paragraph{} It is an error to access the memory pointed to by dst or src until
the copy is locally visible.

\paragraph{} The following two code sequences show the relationship between
\memput{} and \function{}:

\begin{verbatim} 
upc_memput( dst, src, n ); // effects are locally visible after return
...                        // code that may access targets of dst and src
upc_fence;                 
...                        // effects are globally visible, but also all
                           //   prior accesses are globally visible


upc_memput_asynci( dst1, src1, n );
upc_memput_asynci( dst2, src2, n );
...           // code that may not access targets of dst1, dst2, src1, and src2
upc_lsynci(); // effects are locally visible after return (optional call)
...           // code that may access targets of dst1, dst2, src1, and src2 (requires lsynci)
upc_gsynci(); // effects are globally visible after return (optional call)
...
upc_fence;
...           // all prior accesses are globally visible
\end{verbatim}

\newpage
\subsection{upc\_memset\_asynci}
\def\function{{\tt upc\_memset\_asynci}}

\subsubsection{Synopsis}

\begin{verbatim}
void upc_memset_asynci( shared void *dst, int c, size_t n );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The parameters shall be treated identically to the parameters of
\memset{}.

\paragraph{} The \function{} function shall have the same effects as \memset{}.
The effects of all prior \function{} calls shall become locally visible after a
successful \lsynci{} call and shall become globally visible after either a
successful \gsynci{} call or a fence (strict access).

\paragraph{} It is an error to access the memory pointed to by dst until
the copy is locally visible.

\paragraph{} The following two code sequences show the relationship between
\memset{} and \function{}:

\begin{verbatim} 
upc_memset( dst, c, n );   // effects are locally visible after return
...                        // code that may access target of dst
upc_fence;                 
...                        // effects are globally visible, but also all
                           //   prior accesses are globally visible


upc_memset_asynci( dst1, c, n );
upc_memset_asynci( dst2, c, n );
...           // code that may not access targets of dst1 and dst2
upc_lsynci(); // effects are locally visible after return (optional call)
...           // code that may access targets of dst1 and dst2 (requires lsynci)
upc_gsynci(); // effects are globally visible after return (optional call)
...
upc_fence;
...           // all prior accesses are globally visible
\end{verbatim}

\newpage
\subsection{upc\_lsync\_attempt}
\def\function{{\tt upc\_lsync\_attempt}}

\subsubsection{Synopsis}

\begin{verbatim}
int upc_lsync_attempt( upc_handle_t* handle ); 
\end{verbatim}

\subsubsection{Description}

\paragraph{} The handle argument shall point to a valid handle or the value
\complete{}.

\paragraph{} The \function{} function returns 1 if the copy associated with the
handle is locally visible, or 0 otherwise.  If in addition to being locally
visible the copy is globally visible, the \function{} function sets *handle to
\complete{}; otherwise, it leaves *handle unchanged.

\paragraph{} The \function{} function returns 1 if *handle == \complete{}.

\paragraph{} The \function{} function does not have any fence-like behavior.

\subsection{upc\_lsync}
\def\function{{\tt upc\_lsync}}

\subsubsection{Synopsis}

\begin{verbatim}
void upc_lsync( upc_handle_t* handle );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The handle argument shall point to a valid handle or the value
\complete{}.

\paragraph{} The \function{} function does not return until the copy associated
with the handle is locally visible.  If in addition to being locally visible the
copy is globally visible, the \function{} function sets *handle to \complete{};
otherwise, it leaves *handle unchanged.

\paragraph{} The \function{} function returns 1 if *handle == \complete{}.

\paragraph{} The \function{} function does not have any fence-like behavior.

\newpage
\subsection{upc\_lsynci\_attempt}
\def\function{{\tt upc\_lsynci\_attempt}}

\subsubsection{Synopsis}

\begin{verbatim}
int upc_lsynci_attempt( void );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The \function{} function returns 1 if all \_asynci copies are
locally visible, or 0 otherwise.

\paragraph{} The \function{} function does not have any fence-like behavior.

\subsection{upc\_lsynci}
\def\function{{\tt upc\_lsynci}}

\subsubsection{Synopsis}

\begin{verbatim}
void upc_lsynci( void );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The \function{} function does not return until all \_asynci copies
are locally visible.

\paragraph{} The \function{} function does not have any fence-like behavior.

\newpage
\subsection{upc\_gsync\_attempt}
\def\function{{\tt upc\_gsync\_attempt}}

\subsubsection{Synopsis}

\begin{verbatim}
int upc_gsync_attempt( upc_handle_t* handle ); 
\end{verbatim}

\subsubsection{Description}

\paragraph{} The handle argument shall point to a valid handle or the value
\complete{}.

\paragraph{} The \function{} function returns 1 and sets *handle to \complete{}
if the copy associated with the handle is globally visible, or returns 0 and
leaves *handle unchanged otherwise.

\paragraph{} The \function{} function returns 1 if *handle == \complete{}.

\paragraph{} If the \function{} function returns 1, then it is a half-fence in the
upward direction.  A relaxed access issued before the successful \function{} call
may become visible after the call, but a relaxed access issued after the
successful \function{} call may not become visible before the call.

\subsection{upc\_gsync}
\def\function{{\tt upc\_gsync}}

\subsubsection{Synopsis}

\begin{verbatim}
void upc_gsync( upc_handle_t* handle );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The handle argument shall point to a valid handle or the value
\complete{}.

\paragraph{} The \function{} function does not return until the copy associated
with the handle is globally visible.  It sets *handle to \complete{}.

\paragraph{} The \function{} function returns immediately if *handle ==
\complete{}.

\paragraph{} The \function{} function is a half-fence in the upward direction.
A relaxed access issued before a \function{} call may become visible after the
call, but a relaxed access issued after a \function{} call may not become
visible before the call.

\newpage
\subsection{upc\_gsynci\_attempt}
\def\function{{\tt upc\_gsynci\_attempt}}

\subsubsection{Synopsis}

\begin{verbatim}
int upc_gsynci_attempt( void );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The \function{} function returns 1 if all \_asynci copies are
globally visible, or 0 otherwise.

\paragraph{} If the \function{} function returns 1, then it is a half-fence in the
upward direction.  A relaxed access issued before the successful \function{} call
may become visible after the call, but a relaxed access issued after the
successful \function{} call may not become visible before the call.

\subsection{upc\_gsynci}
\def\function{{\tt upc\_gsynci}}

\subsubsection{Synopsis}

\begin{verbatim}
void upc_gsynci( void );
\end{verbatim}

\subsubsection{Description}

\paragraph{} The \function{} function does not return until all \_asynci copies
are globally visible.

\paragraph{} The \function{} function is a half-fence in the upward direction.  A
relaxed access issued before a \function{} call may become visible after the call,
but a relaxed access issued after a \function{} call may not become visible before
the call.

\end{document}
